Papers with multimodal attention model
MSMO: Multimodal Summarization with Multimodal Output (D18-1)
Copied to clipboard
| Challenge: | Existing studies show that multimodal summarization can improve user satisfaction for informativeness of summaries by using information in visual modality. |
| Approach: | They propose a task to generate text and select the most relevant image from the multimodal input and a novel multimodal automatic evaluation method to evaluate multimodal outputs. |
| Outcome: | The proposed method improves user satisfaction by 12.4% compared to the current system . |
Longform Multimodal Lay Summarization of Scientific Papers: Towards Automatically Generating Science Blogs from Research Articles (2024.lrec-main)
Copied to clipboard
| Challenge: | Science blogs and lay-speak are critical to communicating scientific information to the general public and policymakers. |
| Approach: | They propose to use presentation transcripts and slides to generate a scientific blog from a research article in layperson's terms. |
| Outcome: | The proposed approach can generate a blog text and select the most relevant figures to explain a research article in layperson’s terms, essentially a science blog. |